Summary and discussion of: “Why Does Unsupervised Pre-training Help Deep Learning?”

Before getting into how unsupervised pre-training improves the performance of deep architecture, let’s first look into some basics. Let’s start with logistic regression, which is one of the first models for classification that is taught in machine learning. Logistic classification deals with the supervised learning problem of learning a mapping F : X → Y given a set of training points X = {x1 . . .xn} and a set of class labels Y = {y1, . . . , yn} where xi is assigned a class label yi. The mapping is defined by the function p(Y = 1|X) = 1 1+exp (−(WTX+b)) . There is another way of looking at the logistic classifier. One can think of the X as input to a node in a graphical model and the node does two things: it sums up the inputs multiplied by weights of the edges and then applies a sigmoid on the result. A diagram representing such a function is shown in Figure 1. The node that performs the summation and the non-linear transformation is called a neuron. The summation function is called input activation (a(x) = W TX + b) and the non-linear transform (h(x) = g(a(x)) is called output activation of the neuron. Let’s take a look at an example in Figure 2. An AND function can clearly be modelled using a neuron but a XOR function cannot be directly modelled using a single neuron. If